Image Generation

ABSTRACT

A method of generating an image comprises receiving a signal over a USB interface comprising encoded display data for one or more groups of pixels within an image, the encoded display data comprising display data for less than the whole of the image, and position data for the or each group of pixels. The encoded display data is decoded to generate pixel data for each group of pixels. A frame buffer, which may be part of the display, is then updated with the generated pixel data according to the position data ready for output to the display device.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.13/798,309, filed, Mar. 13, 2013, pending, the entirety of which isincorporated herein by reference.

BACKGROUND

This invention relates to a method of generating an image and to adevice for generating an image.

In desktop computing, it is now common to use more than one displaydevice. Traditionally, a user would have a computer with a singledisplay device attached, but now it is possible to have more than onedisplay device attached to the computer, which increases the usabledisplay area for the worker. For example, International PatentApplication Publication WO 2007/020408 discloses a display system whichcomprises a plurality of display devices, each displaying respectivelyan image, a data processing device connected to each display device andcontrolling the image displayed by each display device, and a userinterface device connected to the data processing device. Connectingmultiple display devices to a computer is a proven method for improvingproductivity.

The connection of an additional display device to a computer presents anumber of problems. In general, most computers will be provided withonly one dedicated video output port such as a VGA output connection.One method by which a second display device can be added to a computeris by adding an additional graphics card to the internal components ofthe computer, which will have an additional graphics output port. Theadditional graphics card will provide an additional video output whichwill allow the second display device to be connected to the computer.However, this solution is relatively expensive and is not suitable formany non-technical users of computers, for whom physically correctlyadding the additional video card to their computer is a difficult task.

An alternative method of connecting the second display device is toconnect the additional display device to a USB socket on the computer,as, in general, all modern computers are provided with multiple USBsockets. This provides a simple connection topology for the user, butrequires additional hardware and software to be present, as in general,USB has a bandwidth that makes the provision of a good quality videooutput a non-trivial task. It is also desirable that any additionalhardware between the computer and the display device is kept as simpleas possible, in order to keep down the cost of this solution of addingan additional display device. This means that when connecting anadditional display device using a limited bandwidth technology such asUSB, certain complex tasks can be difficult to achieve. This isparticular the case as desktop computers are more frequently being usedto output video, for example through websites such as www.youtube.com.

SUMMARY

It is therefore an object of the invention to improve upon the knownart.

According to a first aspect of the present invention, there is provideda method of generating an image comprising:

maintaining, at a display control device, a frame buffer having displaydata for an entire image, the display data comprising pixel data foreach pixel of the image;

receiving a signal over a general purpose data connection at the displaycontrol device, the signal comprising encoded display data for one ormore groups of pixels within the image, the encoded display datacomprising display data for less than the whole of the image, andposition data indicating a position of the one or more groups of pixelswithin the image,

decoding the encoded display data to generate pixel data for the one ormore groups of pixels,

updating the frame buffer with the generated pixel data for the one ormore groups of pixels according to the position data for the one or moregroups of pixels, whereby the pixel data for the one or more groups ofpixels provide a partial update of the image, and

outputting at least a portion of the display data stored in the framebuffer to a display over a display-specific data standard connection.

According to a second aspect of the present invention, there is provideda device for generating an image comprising:

a frame buffer configured to store display data for an entire image, thedisplay data comprising pixel data for each pixel of the image;

a receiver arranged to receive a signal over a general purpose dataconnection, the signal comprising display data for one or more groups ofpixels within the image, the encoded display data comprising displaydata for less than the whole of the image, and position data indicatinga position of the one or more groups of pixels within the image,

a processor arranged to decode the encoded display data to generatepixel data for the one or more groups of pixels and to update the framebuffer with the generated pixel data for the one or more groups ofpixels according to the position data for the one or more groups ofpixels, whereby pixel data for the one or more groups of pixels providea partial update of the image, and

an output device arranged to output at least a portion of the displaydata stored in the frame buffer to a display over a display-specificdata standard connection.

According to a third aspect of the present invention, there is provideda system for generating an image comprising:

a computer arranged to encode display data for one or more groups ofpixels within an image and position data indicating a position of eachof the groups of pixels within the image, the display data being lessthan the display data for an entire image, and to transmit an encodedsignal of the encoded display data to a display control device over ageneral purpose data connection; and

a display control device as described above.

According to a fourth aspect of the present invention, there is provideda tangible non-transitory computer readable medium including executableinstructions which, when executed in a processing system, cause theprocessing system to:

maintain, at a display control device, a frame buffer having displaydata for an entire image, the display data comprising pixel data foreach pixel of the image;

receive a signal over a general purpose data connection at the displaycontrol device, the signal comprising display data for one or moregroups of pixels within the image, the encoded display data comprisingdisplay data for less than the whole of the image, and position dataindicating a position of the or one or more groups of pixels within theimage,

decode the encoded display data to generate pixel data for the one ormore groups of pixels,

update the frame buffer with the generated pixel data for the one ormore groups of pixels according to the position data for the one or moregroups of pixels, whereby the one or more groups of pixels provide apartial update of the image, and

output at least a portion of the display data stored in the frame bufferto a display over a display-specific data standard connection.

Owing to the invention, it is possible to provide an effective solutionto the problem of providing video over a restricted bandwidth such asUSB to a level of sufficient quality that the end user will not see anoticeable difference in the appearance of the ultimate image, whencompared to standard techniques carried over dedicated graphics channelssuch as VGA. The device of the invention can be employed, for example,between a computer and an additional display device and will receive aUSB input and provide a VGA output.

A Haar transform is not optimal for typical photographic datacompression but is very appropriate for the compression of arbitraryscreen images for a computer system, for two reasons. The first reasonis that screen compression must be very fast, so that it can be appliedto the entire computer screen many times per second without causingundue load on the computers CPU. The second is that computer screens mayshow photographic information but may also show images such as a page ofwriting or graphical information. In these cases it is very importantthat sharp edges are preserved, even if at the expense of precise colourfidelity. The Haar transform performs well in both these cases.

In one embodiment, the device for generating an image is a decoder fordecoding the encoded display data which uses entropy coding of ACcoefficients and using this to indicate encoding of the parameters of aHaar transform, the Haar transform being of AC coefficients of YCrCbchannels of a visual image with the resulting encoding being receivedover USB by a USB 2.0 or SS USB device, where preferably, the output ofthe device is being protected against copy protection using AES,negotiated using HDPC.

Preferably, AC coefficients are ordered so that trailing coefficientsrepresent highest-frequency data, and trailing zero coefficients may beomitted. The reason for this is that in many typical computer displayimages there are large areas where adjacent pixels are identical, orhave very similar values. The highest frequency information in the Haartransform consists of the difference in value between adjacent pixels,and in practice the values are often zero. By ordering the values toleave these at the end, and omitting trailing zero values, compressionis typically improved.

Advantageously, colour information is represented in the YCrCb colourspace. The reason for this is that typical computer display imagescontain much of their information as changes in brightness (Y) and sothe chroma values (CrCb) tend to be more easily compressible. In imagecompression systems for photographic data (such as JPEG, MPEG) thechroma data is subsampled, but for computer displays it is important notto do this. For instance, it would mean that coloured writing in a wordprocessor or spreadsheet would be blurred by such a subsampling process,which is unacceptable to the user.

The device is structured so that a low compute load is placed on thehost system to encode and transmit the video data with very low latencycaused by the sequence of encoding, transmitting over USB, decoding anddisplaying so that a human user finds the USB-connected display to be anacceptable replacement for a display connected via a dedicated videosignal cable. The video encoded format is suitable both for photographicdata and for computer desktop displays. Existing image compressionapproaches split between approaches which are ideally suited forphotographic or movie data, for example, MPEG and other such codecs,which would not do a good job of computer desktop displays and losslessframe buffer transmission, which performs very poorly when the userplays a movie or displays photographic information.

The decoding method and device of the invention combines these twoapproaches, thereby providing complete fidelity when required fordesktop displays, yet performing reasonably well in the presence ofphotographic information, in a format which is suitable for very lowlatency encoding and decoding. The result does not achieve the very highcompression ratios of codecs such as MPEG, but is better suited thanMPEG for the attachment of visual displays via low cost networks andserial buses.

In a preferred embodiment, the video decoding device (embodied as asilicon chip or an end user product) receives an encoded video signal,decodes the video signal, and outputs the decoded signal to a videodisplay 30 device over a video link such as HOMI, OVI, OP, such that theencoded signal arrives over some variant of the USB interface asstandardised by the USB-IF, and the encoded signal makes use of the Haartransform for each 8×8 pixel square of the source images, in each colourchannel, AC coefficients of each Haar transform are encoded as variablelength bit fields, where the variable encoding is defined using a simpleand compact UseVec form (consistent with the definition in thebackground section). The AC coefficients are ordered so that trailingcoefficients represent highest-frequency data, and trailing zerocoefficients may be omitted; the colour channels are all at the sameresolution, with no subsampling of chroma channels being performed,except as dynamically determined by the number of coefficients providedto the Haar transforms. The encoded signal format is divided into groupsof 8×8 tiles so that partial updates of video frame may be performed.Each tile group separately identifies its intended location in thedisplayed image so that local changes to not require a complete frameupdate. Each tile group contains fewer than twenty 8×8 tiles and thedecoded signal is sent to a video raster display port such as HOMI, OVI,DisplayPort, or is displayed directly on a visual display which formspart of the device.

The functionality described here can be implemented in hardware,software executed by a processing apparatus, or by a combination ofhardware and software. The processing apparatus can comprise a computer,a processor, a state machine, a logic array or any other suitableprocessing apparatus. The processing apparatus can be a general-purposeprocessor which executes software to cause the general-purpose processorto perform the required tasks, or the processing apparatus can bededicated to perform the required functions. Another aspect of theinvention provides machine-readable instructions (software) which, whenexecuted by a processor, perform any of the described methods. Themachine-readable instructions may be stored on an electronic memorydevice, hard disk, optical disk or other machine-readable storagemedium. The machine-readable instructions can be downloaded to thestorage medium via a network connection.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described, by way ofexample only, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram of a display system,

FIG. 2 is a schematic diagram of components of the display system,

FIG. 3 is a schematic diagram of a display control device, and

FIG. 4 is a schematic diagram of an image showing a group of pixeltiles.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

A display system is shown in FIG. 1. The system comprises a processingdevice 10, display devices 12 and user interface devices 14. The userinterface devices are a keyboard 14 a and a mouse 14 b. The system shownin FIG. 1 is a standard desktop computer, with an additional displaydevice 12 b, which is composed of discrete components that are locallylocated but could equally be a device such as a laptop computer orsuitably enabled handheld device such as a mobile phone or pda (personaldigital assistant) all using an additional display 12 b. Similarly, thesystem may comprise part of a networked or mainframe computing system,in which case the processing device 10 may be located remotely from theuser input devices 14 and the display devices 12, or indeed may have itsfunction distributed amongst separate devices.

The display devices 12 show images 16, and the display of the images 16is controlled by the processing device 10. One or more applications arerunning on the processing device 10 and these are represented to theuser by corresponding application windows 18, with which the user caninteract in a conventional manner. A cursor 20 is shown, and the usercan control the movement of the cursor 20 about the images 16 shown onthe display device 12 using the computer mouse 14 b, again in a totallyconventional manner. The user can perform actions with respect to anyrunning application via the user interface device 14 and these actionsresult in corresponding changes in the images 16, displayed by thedisplay device 12.

The operating system run by the processing device 10 uses virtualdesktops to manage the multiple display devices 12. Each physicaldisplay device 12 is represented by a frame buffer that containseverything currently shown on that display device 12. The operatingsystem is configured to arrange these frame buffers into a singlevirtual desktop. When these frame buffers are arranged in the virtualdesktop 22 in the same relative positions in which the physical displaydevices 12 are relatively placed, then the operating system can drawobjects on all the display devices 12 in a natural way. The virtualdesktop is a combination of the respective images 16 a and 16 b beingshown by the display devices 12. If the user moves the mouse 14 a suchthat the cursor 20 moves right off the edge of one display device 12 a,then the cursor 20 appears on the left of the display device 12 b to theright. Similarly a window 18 spread across several display devices 12appears properly lined up between the display devices 12.

More detail of the connection of the secondary display device 12 b tothe processing device 10 is shown in FIG. 2. The processing device 10connects to the secondary display device 12 b via a display controldevice 22. The display control device 22 is connected to the processingdevice 10 via a standard USB connection, and appears to the processingdevice 10 as a USB connected device. Any communications between theprocessing device 10 and the display control device 22 are carried outunder the control of a USB driver specifically for the display controldevice 22. Such devices allow the connection of the secondary displaydevice 12 b to the processing device 10 without the need for anyhardware changes to the processing device 10.

The display control device 22 connects to the display device 12 b via astandard VGA connection, and the display device 12 b is a conventionaldisplay device 12 which requires no adjustment to operate in the displaysystem shown in FIG. 2. As far as the display device 12 b is concerned,it could be connected directly to the graphics card of a processingdevice; it is unaware that the graphical data displayed by the displaydevice 12 b has actually been first sent via a USB connection to anintermediate component, the display control device 22. Multipleadditional display devices 12 can be 15 connected to the processingdevice 10 in this way, as long as suitable USB slots are available onthe processing device 10.

The display control device 22 is external to the processing device 10and is not a graphics card. It is a dedicated piece of hardware thatreceives graphical data via the USB connection from the processingdevice 10 and transforms that graphics data into a VGA format that willbe understood by the display device 12 b. In topological terms USB andVGA are only examples of data standards that can be used to connect theadditional display device 12 b to the processing device 10. The generalprinciple is that a general-purpose data network (such as USB orEthernet) connects the processing device 10 to the display controldevice 22 and a display-specific data standard (such as VGA or DVI) isused on the connection from the display control device 22 to the displaydevice 12 b.

The display control device 22 is shown in more detail in FIG. 3. Thedevice 22 operates to receive encoded display data via a USB input port24, which is processed with a processor 26 that is connected to a framebuffer 28 and provides an output via an output port 30. The device 22 isfor generating the image 16 b that is displayed by the additionaldisplay device 12 b. The device 22 is effectively an image decoder, asit receives encoded display data which is then decoded to pixel datathat is stored within the frame buffer 28. The display control device 22is a lightweight piece of hardware for adding an additional display 12 bto a computer 10.

The incoming signal received by the USB input port 24 comprises encodeddisplay data for one or more rectangular groups of pixel tiles withinthe image 16 b and also position data for each of the rectangular groupsof pixel tiles. This display data is less than the display data for anentire image, and reduces the amount of data that has to be transmittedbetween the computer 10 and the display control device. A pixel tilecomprises an 8×8 array of pixels and the group size could be sixteenpixel tiles, for example, in a 4×4 array of tiles. Essentially only anupdate of part of the image 16 b is being sent.

The processor 26 takes the encoded tiles and decodes these tiles intopixel data and updates the frame buffer 28 with the new decoded pixeldata according to the position data in the received signal. In this way,the frame buffer 28 will always maintain pixel data for the entire image16 b, and this pixel data is continuously being updated by the processor26. The processor 26 will then output at least a portion of the framebuffer contents via the output port 30 to the display device 12 b overthe VGA connection. The display control device 22 is configured as adedicated piece of hardware for the decoding process.

FIG. 3 shows one embodiment of the display control device 22, whichincludes a dedicated frame buffer 28. In other solutions, the framebuffer 28 may not be part of the device 22, but may be present onanother separate chip, in which case the processor 26 writes to thatframe buffer 28 via a 25 suitable external output port. The device 22may not be a separate device as shown in FIG. 2, but may form part ofthe actual display device 12. In this case, the output device 30 is notan output port as such, but is the actual display device 12, which iswritten directly from the frame buffer 28.

FIG. 4 illustrates the concept of a group 32 of pixel tiles 34 within animage 16 b. The image 16 b is the image that is being displayed by thesecondary display device 12 b. This image 16 b is stored in the framebuffer 28 of the display controller 22. To update this image 16 b, thenone or more groups 32 of pixel tiles 34 is transmitted via the USBconnection to the display control device 22. The display control device22 receives the signal comprising the encoded display data, decodes thatdata and updates the frame buffer 28 accordingly. The frame buffer 28 isthen read out via the port 30 to the display device 12 b.

In FIG. 4 only a single group 32 of pixel tiles 34 is shown. The systemcan be configured so that the number of tiles 34 within a group 32 isflexible, but generally it is desirable to limit the number of tiles 34in a group 32 to no more than twenty. The group 34 can be locatedanywhere within the image 16 b, and hence the need to include positioninformation in the signal between the computer 10 and the displaycontrol device 22. The position information is used by the device 22,when writing the decoded pixel data into the frame buffer 28. Only therelevant part of the frame buffer 28 is updated.

Although it is not necessarily the case that all of the tiles 34 withinthe group 32 have actually changed from one frame to another, thestructure of the display data in the signal that is carried from thecomputer 10 to the display control device 22 is such that all of thetiles 34 within the group 32 must be encoded and transmitted to thedisplay control device 22. The selection of the position of the group 32in the image 16 b is made by the computer 10, on the 20 basis ofcapturing the most pixel tiles 34 that have changed between twoconsecutive frames, in the most efficient manner.

The operation of the device 22 is optimised for low latency, high visualfidelity, and low encoding computation effort. These requirements aredriven by the application of forwarding video displays including desktopdisplays and more dynamic desktop video. When a desktop display is usedcomplete or near-complete fidelity to the original source material isrequired. When video information is presented, greater loss may beintroduced. The result does not achieve the very high compression ratiosof codecs such as MPEG, but is better suited than MPEG for theattachment of visual displays via low cost networks and serial buses.

The frame buffer 28 is designed for storing the image 16 b so that thevideo output port 30 can refresh the display 12 b without any trafficover the USB input port 24. The frame buffer 28 can be in any internalformat; may be in the data input format, or a rectangular pixel map inRGB or YCrCb, or some other format. The frame buffer 28 may bedouble-buffered or triple buffered to reduce tearing when rapidlyupdating the display. The frame buffer 28 may be on-chip memory, orexternal memory chips such as some DRAM technology.

The video decoding device 22 (as a silicon chip or an end user product)receives an encoded video signal, decodes the video signal, and outputsthe decoded signal to a video display device 12 b over a video link suchas HOMI, OVI, OP. The encoded signal arrives over a USB interface asstandardised by the USB-IF and the encoded signal format is divided intorectangular groups of 8×8 pixel tiles with each tile group including anexplicit screen position, so that partial updates of the video frame maybe performed. Each tile group separately identifies its intendedlocation in the displayed image so that local changes to the displayedimage do not require a complete frame update. Each 15 tile groupcontains fewer than twenty 8×8 tiles.

The encoded signal makes use of the Haar transform for each 8×8 pixelsquare of the source images, in each colour channel. AC coefficients ofeach Haar transform are encoded as variable length bit fields, where thevariable encoding is defined using a simple and compact entropy form.The decoded signal is sent to a video raster display port such as HOMI,OVI, DisplayPort, or is displayed directly on a visual display whichforms part of the device. The device 22 refreshes the image as requiredat the video link port by means of a frame buffer which stores thecurrent image in some suitable format.

Although an embodiment has been described in detail above, it will beappreciated that various changes, modifications and improvements can bemade by a person skilled in the art without departing from the scope ofthe present invention as defined in the claims.

1. A method of generating an image comprising: maintaining, at a displaycontrol device, a frame buffer having display data for an entire image,the display data comprising pixel data for each pixel of the image;receiving a signal over a general purpose data connection at the displaycontrol device, the signal comprising encoded display data for one ormore groups of pixels within the image, the encoded display datacomprising display data for less than the whole of the image, andposition data indicating a position of the one or more groups of pixelswithin the image, decoding the encoded display data to generate pixeldata for the one or more groups of pixels, updating the frame bufferwith the generated pixel data for the one or more groups of pixelsaccording to the position data for the one or more groups of pixels,whereby the pixel data for the one or more groups of pixels provide apartial update of the image, and outputting at least a portion of thedisplay data stored in the frame buffer to a display over adisplay-specific data standard connection.
 2. A method according toclaim 1, wherein a group of pixels comprises a group of pixel tiles,wherein a pixel tile comprises an 8×8 array of pixels.
 3. A methodaccording to claim 2, wherein a group of pixel tiles comprises fewerthan twenty pixel tiles.
 4. A method according to claim 1, whereindecoding the encoded display data comprises obtaining a coefficient forthe one or more groups of pixels from the encoded display data byconverting variable length bit fields in the encoded display data intoAC coefficients.
 5. A method according to claim 1, wherein the generalpurpose data connection comprises a USB connection or an Ethernetconnection.
 6. A method according to claim 1, wherein thedisplay-specific data standard connection comprises a VGA, HDMI, DVI orDisplayPort connection.
 7. A method according to claim 1, wherein theone or more groups of pixels are located anywhere within the image.
 8. Amethod according to claim 1, wherein the position data separatelyidentifies the intended location of the one or more groups of pixels inthe frame buffer.
 9. A device for generating an image comprising: aframe buffer configured to store display data for an entire image, thedisplay data comprising pixel data for each pixel of the image; areceiver arranged to receive a signal over a general purpose dataconnection, the signal comprising display data for one or more groups ofpixels within the image, the encoded display data comprising displaydata for less than the whole of the image, and position data indicatinga position of the one or more groups of pixels within the image, aprocessor arranged to decode the encoded display data to generate pixeldata for the one or more groups of pixels and to update the frame bufferwith the generated pixel data for the one or more groups of pixelsaccording to the position data for the one or more groups of pixels,whereby pixel data for the one or more groups of pixels provide apartial update of the image, and an output device arranged to output atleast a portion of the display data stored in the frame buffer to adisplay over a display-specific data standard connection.
 10. A deviceaccording to claim 9, wherein a group of pixels comprises a group ofpixel tiles, wherein a pixel tile comprises an 8×8 array of pixels. 11.A device according to claim 10, wherein a group of pixel tiles comprisesfewer than twenty pixel tiles.
 12. A device according to claim 9,wherein the processor is configured, when decoding the encoded displaydata, to obtain a coefficient for the one or more groups of pixels fromthe encoded display data by converting variable length bit fields in theencoded display data into AC coefficients.
 13. A device according toclaim 9, wherein the receiver comprises a USB port or an Ethernet port.14. A device according to claim 9, wherein the display-specific datastandard connection comprises a VGA, HDMI, DVI or DisplayPortconnection.
 15. A device according to claim 9, wherein the one or moregroups of pixels are located anywhere within the image.
 16. A deviceaccording to claim 9, wherein the position data separately identifiesthe intended location of the one or more groups of pixels in the framebuffer.
 17. A system comprising the device of claim 9, and furtherincluding: a computer arranged to encode the display data for the one ormore groups of pixels within the image and the position data indicatingthe position of each of the groups of pixels within the image, thedisplay data being less than the display data for an entire image, andto transmit an encoded signal of encoded display data to the device overthe general purpose data connection.
 18. A tangible non-transitorycomputer readable medium including executable instructions which, whenexecuted in a processing system, cause the processing system to:maintain, at a display control device, a frame buffer having displaydata for an entire image, the display data comprising pixel data foreach pixel of the image; receive a signal over a general purpose dataconnection at the display control device, the signal comprising displaydata for one or more groups of pixels within the image, the encodeddisplay data comprising display data for less than the whole of theimage, and position data indicating a position of the or one or moregroups of pixels within the image, decode the encoded display data togenerate pixel data for the one or more groups of pixels, update theframe buffer with the generated pixel data for the one or more groups ofpixels according to the position data for the one or more groups ofpixels, whereby the one or more groups of pixels provide a partialupdate of the image, and output at least a portion of the display datastored in the frame buffer to a display over a display-specific datastandard connection.
 19. A tangible computer readable medium accordingto claim 18, wherein a group of pixels comprises a group of pixel tiles,wherein a pixel tile comprises an 8×8 array of pixels.
 20. A tangiblecomputer readable medium according to claim 18, wherein the executableinstructions that cause the processing system to decode the encodeddisplay data comprises instructions that cause the processing system toobtain a coefficient for the one or more groups of pixels from theencoded display data by converting variable length bit fields in theencoded display data into AC coefficients.